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ABSTRACT 

In  the  past  ten  years,  in-air  acoustic  sensor  systems  have  been  developed  for  the  localization  of  small  arms 
fire.  A  number  of  countries  have  systems  at  different  stages  of  advancement  that  they  have  deployed  on 
operations.  These  systems  are  being  mounted  on  vehicles  to  provide  crews  with  better  situational 
awareness,  thereby  enhancing  crew  safety  and  force  protection.  The  acoustic  system  is  starting  to  be 
incorporated  into  vehicle  equipment  as  a  standard  item,  especially  in  new  vehicles.  The  introduction  of 
this  new  technology >  requires  a  pragmatic  approach  to  specifying  the  performance  of  these  systems.  For  a 
number  of  years,  Defence  R&D  Canada  -  Valcartier  has  worked  on  the  development  of  the  Ferret  small 
arms  detection  and  localization  system.  Our  involvement  in  the  design  and  fielding  of  the  Ferret  has 
provided  us  with  an  inside  view  on  how  these  systems  should  be  specified.  In  this  paper,  we  propose  a 
basis  for  the  specification  of  such  a  system. 


1.0  INTRODUCTION 

As  major  buyers  of  military  technology,  governments  are  playing  a  lead  role  in  establishing  performance 
criteria  and  benchmarks  for  evaluation  and  qualification  purposes.  Defence  R&D  Canada,  with  its  long¬ 
term  expertise  in  atmospheric  acoustics,  is  proposing  performance  metrics  for  the  qualification  of  acoustic 
small  arms  localization  systems.  This  consistent  framework  is  a  consolidation  of  criteria  that  have  been 
developed  for  the  procurement  of  the  Ferret  system  by  the  Canadian  Army.  Consistent  performance 
metrics  will  contribute  to  an  improved  understanding  of  in-air  acoustic  technology  by  both  the  public  and 
private  sectors,  especially  for  those  agencies  with  little  knowledge  in  the  field.  In  the  long  run,  this  will 
manage  expectations  at  all  levels  of  the  procurement  process  and  promote  the  strengths  of  the  technology 
and  awareness  of  its  limitations  in  the  best  interests  of  the  troops  in  the  field. 

An  acoustic  small  arms  localization  system1'7  provides  the  location  of  the  shooter  in  range  and  angle,  the 
bullet  miss  distance  and  the  bullet  calibre1.  The  performance  metrics  indicate  the  probability  that  the  shot 
will  be  detected  and  identified,  the  accuracy  of  the  solution  provided  and  the  level  of  confidence  in  that 
accuracy.  There  is  a  dramatic  difference  between  a  system  detecting  90%  of  the  shots  fired  and  providing 
good  shooter  information  on  only  1 0%  of  the  shots,  and  a  system  detecting  only  1 0%  of  the  shots  fired  but 
providing  good  shooter  information  on  90%  of  them.  Providing  the  probability  of  detection  alone  is 
incomplete;  an  operator  needs  to  know  what  level  of  confidence  he  or  she  can  have  in  the  information 
provided. 

The  specific  characteristic  of  in-air  acoustics  is  the  non-stationary  nature  of  the  atmosphere.  Wind, 
turbulence  and  changes  in  the  speed  of  sound  with  height  affect  the  propagation  of  a  sound.  The 
atmosphere  can  enhance  or  impede  the  detection  of  a  sound  for  a  few  seconds  or  a  few  hours.  Terrain  and 


1  See  also  the  Pilar  system  from  Metravib,  France  on  the  Internet. 

Bedard,  J.  (2006)  Performance  Metrics  for  Acoustic  Small  Arms  Localization  Systems.  In  Battlefield  Acoustic  Sensing  for  ISR 
Applications  (pp.  24-1  -  24-10).  Meeting  Proceedings  RTO-MP-SET-107,  Paper  24.  Neuilly-sur-Seine,  France:  RTO. 
Available  from:  http://www.rto.nato.int/abstracts.asp. 
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obstructions  in  the  area  also  affect  the  detection  and  recognition  of  an  acoustic  event.  In  principle,  the 
probabilities  provided  in  performance  metrics  should  be  obtained  from  long-term  averages  taken  in  a 
variety  of  environments.  For  obvious  reasons,  it  is  not  feasible  to  test  small  arms  fire  in  many 
environments.  Even  if  this  were  possible,  it  is  not  desirable  to  select  performance  metrics  that  could  not 
be  used  with  reasonable  effort. 

The  performance  metrics  that  are  proposed  in  this  document  were  derived  for  open  terrain,  an  environment 
typical  of  most  trial  sites  around  the  world.  The  metrics  are  intended  for  single  array  systems  as  opposed 
to  distributed  sensor  networks  that  use  triangulation  and/or  fusion  with  other  sensors  for  detection  and 
recognition.  To  understand  the  compromises  that  were  made  in  selecting  the  performance  parameters, 
Section  2  describes  the  acoustic  phenomenology  of  small  arms  fire.  It  describes  the  sounds  that  are 
created  by  small  arms  fire  and  the  effect  of  the  atmosphere  on  them.  In  Section  3,  we  present  the  proposed 
metrics  along  with  the  reasoning  that  substantiate  the  choice  of  each  parameter.  Section  4  summarizes  the 
approach. 


2.0  ACOUSTIC  PHENOMENOLOGY 

2.1  Propagation  geometry 

Most  small  arms  fire  generates  two  distinct  impulse  sounds:  the  muzzle  blast  and  the  shockwave  (see 
Figure  1).  The  muzzle  blast  is  created  at  the  weapon  by  the  rapid  discharge  of  the  propellant  and  the  fast 
combustion  that  occurs  when  the  unbumed  part  of  the  propellant  mixes  with  air  outside  the  muzzle.  The 
shockwave  is  created  by  the  supersonic  bullet  travelling  through  the  air,  similar  to  the  sonic  boom  of  a 
supersonic  aircraft. 

For  a  sensor  exposed  to  small  arms  fire,  the  muzzle  blast  wave  originates  from  a  point  source  (the 
muzzle),  and  propagates  spherically  from  the  muzzle  at  the  speed  of  sound.  Hence,  it  can  be  detected 
from  anywhere  around  the  firing  position  (shown  in  blue  below).  From  the  sensor  perspective,  the  muzzle 
blast  wave  propagates  from  the  weapon  directly  towards  the  sensor,  given  that  there  are  no  obstructions  in 
the  propagation  path. 


Figure  1:  Muzzle  blast  and  shockwave  propagation  at  low  angle  of  fire  (U) 
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Figure  2:  Muzzle  blast  and  shockwave  propagation  at  high  angle  of  fire  (U) 


The  shockwave  cone  produced  by  the  bullet  is  similar  to  the  triangular  wake  created  by  a  boat  on  the 
surface  of  the  water.  The  tip  of  the  cone  moves  along  the  line  of  fire  at  the  speed  of  the  bullet,  but  the 
acoustic  wave  front  that  is  produced  propagates  perpendicular  to  the  shockwave  front  (in  red)  at  the  speed 
of  sound. 

From  a  sensor  perspective,  the  bullet  travels  from  the  muzzle  of  the  weapon  to  the  point  where  the 
shockwave  detaches  from  the  bullet  and  propagates  to  the  sensor.  The  time  for  the  shockwave  to  reach  the 
sensor  after  the  shot  is  calculated  by  adding  the  bullet’s  time  of  travel  to  the  point  of  detachment  and  the 
propagation  time  of  the  shockwave  from  the  point  of  detachment  to  the  sensor.  As  shown  in 
Figure  2,  at  a  high  angle  of  fire,  the  point  of  detachment  of  the  shockwave  is  close  to  the  weapon,  and  the 
shockwave  and  muzzle  blast  appear  to  originate  from  the  same  area.  From  a  sensor  perspective,  their 
times  of  arrival  are  almost  the  same  and  their  directions  of  arrival  are  almost  the  same. 

As  the  angle  of  fire  and  the  distance  between  the  weapon  and  sensor  vary,  the  difference  between  the 
times  of  arrival  of  the  two  pulses  and  the  difference  between  the  directions  of  arrival  of  the  pulses  change 
in  a  predictable  manner.  This  predictability  allows  the  system  to  map  out  the  entire  geometry  of  the  fire 
event  by  collecting  and  analyzing  the  sounds  produced  by  the  two  pulses.  These  pulses  are  the  only 
sources  of  information  available  to  an  acoustic  system. 

2.2  Acoustic  signal 

Figure  3  illustrates  the  idealized  shockwave  and  muzzle  blast  waves  that  would  be  received  by  a 
microphone  exposed  to  small  arms  fire.  This  idealized  illustration  does  not  allow  for  the  effects  of 
reflections  from  obstructions  and  terrain  features,  reverberation,  atmospheric  refraction  bending  the 
propagation  path  and  filtering  out  the  high  frequencies,  and  diminished  sound  clarity  caused  by 
turbulence. 


RTO-MP-SET-107 


24-3 


UNCLASSIFIED/UNLIMITED 


UNCLASSIFIED/UNLIMITED 


Performance  Metrics  for  Acoustic  Small  Arms  Localization  Systems 


ORGANIZATION 


Figure  3:  Idealized  sound  received  at  a  microphone  exposed  to  small  arms  fire  (U) 
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Figure  4:  Sound  generated  by  small  arms  fire  (from  top  to  bottom):  the  amplitude  of  the  entire 
signal,  the  instant  power  of  the  entire  signal,  the  zoomed  shockwave  region  and  the  zoomed 

muzzle  blast  region 


Figure  4  is  an  example  of  the  sound  measured  by  a  microphone  in  an  actual  small  arms  fire  event.  The  top 
trace  shows  the  entire  sound  pressure  over  a  period  of  0.2  second,  including  the  shockwave  and  muzzle 
blast.  The  second  trace  is  the  instant  power  obtained  by  squaring  the  pressure,  limiting  its  minimum  value 
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for  visual  purposes  and  taking  the  logarithm.  The  other  two  traces  are  segments  of  the  pressure  zoomed 
around  the  shockwave  and  muzzle  blast.  This  particular  case  shows  four  reflections  of  the  shockwave 
and  one  reflection  of  the  muzzle  blast.  It  is  interesting  to  note  the  level  of  noise  created  by  the  shockwave 
reverberation  observed  after  the  arrival  of  the  shockwave.  That  noise  can  remain  for  a  few  seconds  after 
the  arrival  of  the  shockwave.  In  this  particular  example,  the  shot  was  fired  into  an  open  corridor  bordered 
on  each  side  by  trees.  The  reflections  observed  are  reflections  from  the  ground  and  trees,  which  act  like 
walls. 

The  function  of  the  acoustic  system  is  to  recognize  the  primary  shockwave  and  its  associated  muzzle  blast 
pulses  among  the  variety  of  sound  pulses  generated  by  a  shot.  The  system  must  disregard  reflections  and 
reverberations,  as  well  as  the  sound  of  the  bullet  impacting  the  target,  if  detected.  In  addition  to  these 
unwanted  sounds,  the  environment  generates  its  own  sounds  which  must  also  be  disregarded:  lightning, 
doors  slamming,  people  talking,  motors  starting,  etc. 


3.0  PERFORMANCE  METRICS 
3.1  General 

The  following  nine  (9)  parameters  are  proposed  as  the  performance  metrics  for  the  evaluation  and/or 
qualification  of  acoustic  small  arms  localization  systems. 

1 .  Probability  of  detection  as  a  function  of  miss  distance 

2.  Probability  of  full  localization 

3.  Probability  of  good  solution 

4.  Probability  of  false  alarm 

5.  Azimuth  angle  accuracy 

6.  Elevation  angle  accuracy 

7.  Range  accuracy 

8.  Miss  distance  accuracy 

9.  Calibre  accuracy 

The  performance  metrics  are  intended  for  single  array  systems  exposed  to  single  shots.  Multiple  shots 
fired  from  different  shooters  or  by  the  same  shooter  can  be  considered  as  a  sequence  of  single  shots  if  the 
time  between  the  arrivals  of  the  shockwaves  is  more  than  two  seconds.  This  ensures  that  the  muzzle  blast 
of  a  shot  arrives  before  the  shockwave  from  the  next  shot.  This  avoids  the  confusion  in  processing  which 
can  cause  errors  in  the  range  estimate. 

In  addition  to  creating  confusion,  multiple  shots  raise  the  level  of  noise  in  the  environment.  When  a  single 
shot  is  fired,  the  reverberation  noise  (similar  to  the  echo)  can  be  heard  for  as  much  as  10  seconds.  If 
another  shot  is  fired  during  that  time,  the  chances  of  detecting  that  shot  are  lowered  by  the  increased 
background  noise  level.  This  does  not  mean  that  the  acoustic  system  will  stop  working;  it  simply  means 
that  its  performance  might  be  affected  for  a  few  seconds.  When  one  measures  systems  specifications 
using  the  proposed  metrics,  sufficient  time  should  be  allowed  for  the  reverberation  to  fade  away. 
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3.2  Probability  of  detection 

Probability  of  detection  refers  to  the  probability  that  a  shot  will  be  detected  and  recognized.  As  we 
indicated  in  the  previous  section,  a  small  arms  fire  event  is  characterized  by  the  presence  of  the  shockwave 
and  muzzle  blast  pulses.  Muzzle  blast  is  basically  an  explosion.  Several  sources  of  sound  can  produce  a 
similar  pulse:  blank  ammunition,  artillery  simulators,  firecrackers,  etc.  In  addition,  as  the  distance 
increases  between  the  shooter  and  the  sensor,  the  effects  of  turbulence  and  refraction  gradually  modify  the 
time-domain  features  of  the  pulse  and  thus  its  ability  to  be  recognized. 

On  the  other  hand,  the  shockwave  pulse  is  a  sound  that  is  not  common  in  nature  and  is  difficult  to 
reproduce.  For  that  reason,  the  probability  of  detection  is  linked  directly  to  the  probability  of  detecting  the 
presence  of  the  shockwave.  The  shockwave  presence  is  not  the  only  means  of  recognizing  that  a  shot  has 
been  fired,  but  it  is  the  primary  signal  used  for  recognition.  Because  of  the  effects  of  atmospheric 
propagation,  shockwave  recognition  depends  largely  on  the  bullet  miss  distance.  On  average,  as  the  miss 
distance  increases,  the  probability  of  detection  decreases.  That  does  not  preclude  the  fact  that  a 
shockwave  would  be  recognized  from  200  metres  on  one  day  and  not  recognized  from  50  metres  on 
another  day. 

Because  of  the  relation  between  bullet  miss  distance  and  probability  of  detection,  we  propose  that 
probability  of  detection  be  specified  at  a  number  of  given  miss  distances,  such  as  2,  20  and  200  metres. 
The  probability  at  intermediate  miss  distances  may  be  obtained  using  a  simple  linear  (or  possibly  log- 
linear)  interpolation. 

3.3  Probability  of  full  localization 

Probability  of  full  localization  refers  to  the  probability  that  a  complete  solution  will  be  determined,  given 
that  the  shot  has  been  detected.  A  complete  solution  includes  range,  azimuth  and  elevation  angles,  miss 
distance  and  calibre.  The  probability  that  a  system  will  provide  a  full  localization  is  linked  to  the 
probability  of  recognizing  the  muzzle  blast  amongst  the  reverberation,  reflections  and  ambient  sounds.  In 
principle,  the  strength  and  fidelity  of  the  muzzle  blast  depends  on  the  distance  between  the  shooter  and  the 
sensor.  However,  our  experience  showed  that  atmospheric  refraction  and  turbulence  play  a  more 
important  role  than  range  for  distances  greater  than  approximately  200  metres.  Because  of  the 
randomness  nature  of  these  two  effects  (range  and  turbulence),  the  range  dependency  is  not  generally 
significant.  For  that  reason,  a  single  value  could  be  used  for  the  probability  of  full  localization, 
independent  of  range. 

3.4  Probability  of  good  solution  or  confidence  interval 

Probability  of  good  solution  refers  to  the  probability  that  a  solution  provided  by  the  system  will  fall  within 
the  accuracy  bounds  specified  by  the  azimuth  angle  accuracy,  elevation  angle  accuracy,  range  accuracy, 
miss  distance  accuracy  and  calibre  accuracy.  For  example,  a  probability  of  good  solution  of  90%  with  an 
azimuth  angle  accuracy  of  5  degrees,  an  elevation  angle  accuracy  of  10  degrees,  a  range  accuracy  of  10%, 
a  miss  distance  accuracy  of  30%  and  a  calibre  accuracy  of  40%,  indicates  that  the  solution  provided  has  a 
90%  chance  that  the  azimuth  angle  error  be  less  than  5  degrees,  the  elevation  angle  error  be  less  than 
10  degrees,  the  range  error  be  less  than  10%,  the  miss  distance  error  be  less  than  30%  and  the  calibre  error 
be  less  than  40%. 

The  probability  of  good  solution  can  be  estimated  from  a  number  of  live  fire  shots  by  dividing  the  number 
of  shots  that  fell  within  the  accuracy  bounds  for  azimuth  angle,  elevation  angle,  range,  miss  distance  and 
calibre  by  the  total  number  of  shots  fired. 
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The  accuracies  specified  for  azimuth  angle,  elevation  angle,  range,  miss  distance  and  calibre  define  the 
good  solution. 

3.5  Probability  of  false  alarm 

The  probability  of  false  alarm  is  one  piece  of  information  often  requested  by  operators.  It  tells  them  the 
chances  of  being  wrongly  alerted  to  small  arms  fire.  False  alarms  are  caused  by  a  specific  source  of  sound 
that  mimics  the  shockwave  sound.  The  question  then  reverts  to  what  are  the  chances  of  being  in  the 
presence  of  that  source.  For  example,  if  the  system  triggered  to  the  sound  of  thunder,  and  there  were  five 
occurrences  of  that  sound  in  one  year  and  no  small  arms  fire,  the  proportion  of  false  alarms  in  that  year 
would  be  1 00%.  On  the  other  hand,  if  the  system  was  exposed  to  1 00  small  arms  shots,  the  proportion  of 
false  alarms  would  be  5%. 

One  common  source  of  false  alarms  occurs  when  a  shot  is  fired  away  from  the  acoustic  sensor.  In  other 
words,  the  acoustic  system  is  located  behind  the  shooter.  At  that  location,  only  the  muzzle  blast  is 
detected.  There  is  no  shockwave  propagation  behind  the  shooter.  Because  the  muzzle  blast  is  loud  and 
short  in  duration,  some  systems  wrongly  recognize  that  muzzle  blast  as  a  shockwave.  Some  might  argue 
that  this  is  not  a  false  alarm  because  a  shot  was  actually  fired.  We  believe  that  these  systems  should 
exclude  friendly  fire.  This  also  includes  all  firing  in  the  close  vicinity  of  the  sensor.  Even  if  there  were  a 
shockwave  detected  at  the  sensor,  if  the  shooter  is  within  a  certain  range  threshold,  say  5  or  1 0  metres,  the 
event  should  not  trigger  the  acoustic  system. 

The  recognition  of  a  small  arms  fire  event  is  carried  out  through  a  complex  set  of  thresholds,  criteria  and 
rules.  For  some  detection  systems,  a  simple  threshold  can  be  applied  to  the  random  input  noise  to  meet  a 
given  false  alarm  rate.  That  is  not  the  case  for  the  recognition  of  small  arms  fire. 

In  our  opinion,  if  certain  sources  of  noise  create  false  alarms,  the  operators  should  know  about  them.  If  a 
probability  of  false  alarm  is  specified,  a  method  must  also  be  provided  to  verify  that  number;  otherwise  the 
specification  is  meaningless  and  should  not  be  used  within  performance  metrics. 

3.6  Azimuth  angle  accuracy 

Azimuth  angle  accuracy  is  a  bound  that  defines  the  range  of  azimuth  angles  for  a  good  solution.  It  is  used 
in  conjunction  with  the  probability  of  good  solution.  For  example,  an  azimuth  angle  accuracy  of  5  degrees 
and  a  probability  of  good  solution  of  90%  indicate  that  if  the  system  provides  a  solution,  the  azimuth  angle 
error  will  be  less  than  5  degrees  90%  of  the  time. 

Azimuth  angle  accuracy  depends  mostly  on  the  level  of  turbulence  at  the  time  of  the  shot.  The  azimuth 
angle  is  obtained  from  the  direction  of  arrival  of  the  acoustic  muzzle  blast  pulse.  A  turbulent  atmosphere 
often  gives  rise  to  multi-path  propagation,  which  creates  several  acoustic  rays  that  recombine  at  the  sensor. 
This  not  only  alters  the  direction  of  propagation  but  also  decreases  the  coherence  of  the  pulse  between  the 
microphones  within  the  acoustic  array.  A  side  effect  of  turbulence  is  to  decrease  the  similarity  between 
the  sounds  measured  at  each  microphone.  This  decreased  similarity  introduces  errors  in  the  estimation  of 
the  difference  in  the  times  of  arrival,  which  in  turn  translates  into  errors  in  the  estimation  of  the  azimuth. 

3.7  Elevation  angle  accuracy 

Elevation  angle  accuracy  is  a  bound  that  defines  the  range  of  elevation  angles  for  a  good  solution.  It  is 
used  in  conjunction  with  the  probability  of  good  solution.  For  example,  an  elevation  angle  accuracy  of 
1 0  degrees  and  a  probability  of  good  solution  of  90%  indicate  that  if  the  system  provides  a  solution,  the 
elevation  angle  error  will  be  less  than  10  degrees  90%  of  the  time. 
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The  elevation  angle  of  the  shooter  is  obtained  from  the  direction  of  propagation  of  the  muzzle  blast. 
Propagation  of  the  muzzle  blast  wave  is  strongly  affected  by  the  atmosphere.  The  changes  in  the  speed  of 
sound  with  height  caused  by  wind  and  the  changing  ground  temperature  during  the  day  cause  refraction, 
which  bends  the  acoustic  propagation  path  downward  or  upward.  In  other  words,  sounds  do  not  propagate 
in  a  straight  line.  They  either  propagate  up  and  then  down  like  a  ball,  or  down  and  then  up.  Note  that  the 
azimuth  angle  is  not  affected  by  atmospheric  refraction,  only  the  elevation  angle. 

The  atmosphere  bends  the  acoustic  ray  to  some  extent,  but  elevation  angle  estimates  are  still  valid  at  short 
ranges. 


3.8  Range  accuracy 

Range  accuracy  is  a  bound  that  defines  the  range  of  shooter  distances  (range)  for  a  good  solution.  It  is 
used  in  conjunction  with  the  probability  of  good  solution.  For  example,  a  range  accuracy  of  10%  and  a 
probability  of  good  solution  of  90%  indicate  that  if  the  system  provides  a  solution,  then  90%  of  the  time 
the  range  error  will  be  less  than  10%.  Range  accuracy  is  relative  to  the  range  estimate  and  thus  is 
expressed  as  a  percentage.  Unlike  azimuth  error,  range  error  increases  almost  linearly  with  range.  This  is 
due  mainly  to  errors  in  the  assumed  ballistics  of  the  bullet. 

Range  accuracy  depends  on  the  geometry  of  the  shooting,  the  accuracy  in  the  measurement  of  the 
direction  and  time  of  arrival  of  the  shockwave  and  muzzle  blast,  the  time  feature  of  shockwave  pulse,  and 
advance  knowledge  of  the  ballistics  of  the  bullet.  The  estimation  of  range  requires  the  reconstruction  of 
the  entire  geometry  of  shooting.  When  the  distance  between  the  shooter  and  the  shockwave  detach  point 
is  small,  the  most  important  parameters  in  range  estimation  are  the  difference  in  the  angles  of  arrival  of  the 
muzzle  blast  and  shockwave  and  the  time  features  of  the  shockwave:  duration  and  amplitude.  At  the  other 
extreme,  when  the  distance  between  the  shooter  and  the  shockwave  detach  point  is  large  compared  to  the 
miss  distance,  the  most  important  parameter  in  range  estimation  is  advance  knowledge  of  the  bullet  drag. 

3.9  Miss  distance  accuracy 

Miss  distance  accuracy  is  a  bound  that  defines  the  range  of  miss  distances  for  a  good  solution.  It  is  used  in 
conjunction  with  the  probability  of  good  solution.  For  example,  a  miss  distance  accuracy  of  30%  and  a 
probability  of  good  solution  of  90%  indicate  that  if  the  system  provides  a  solution,  then  90%  of  the  time 
the  miss  distance  error  will  be  less  than  30%.  Miss  distance  accuracy  is  relative  to  the  miss  distance 
estimate  and  thus  is  expressed  as  a  percentage. 

Miss  distance  accuracy  is  linked  directly  to  the  time  feature  of  the  shockwave.  The  shooter  range  and  the 
ballistics  of  the  bullet  have  a  weak  effect  on  those  parameters  by  changing  the  speed  of  the  bullet  at  the 
detach  point,  which  in  turn  changes  the  duration  and  amplitude  of  the  shockwave.  In  order  to  simplify  the 
performance  metrics,  miss  distance  accuracy  is  not  expressed  in  terms  of  range  or  bullet  calibre.  The 
value  provided  for  accuracy  should  take  these  effects  into  account. 

3.10  Calibre  accuracy 

Calibre  accuracy  is  a  bound  that  defines  the  range  of  calibres  for  a  good  solution.  It  is  used  in  conjunction 
with  the  probability  of  good  solution.  For  example,  a  calibre  accuracy  of  30%  and  a  probability  of  good 
solution  of  90  %  indicate  that  if  the  system  provides  a  solution,  then  90%  of  the  time  the  calibre  error  will 
be  less  than  30%.  Calibre  accuracy  is  relative  to  the  calibre  estimate  and  thus  is  expressed  as  a  percentage. 
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Calibre  accuracy  is  linked  directly  to  the  time  feature  of  the  shockwave.  The  shooter  range  and  the 
ballistics  of  the  bullet  have  a  weak  effect  on  those  parameters  by  changing  the  speed  of  the  bullet  at  the 
detach  point,  which  in  turn  changes  the  duration  and  amplitude  of  the  shockwave.  In  order  to  simplify  the 
performance  metrics,  miss  distance  accuracy  is  not  expressed  in  terms  of  range  or  bullet  calibre.  The 
value  provided  for  accuracy  should  take  these  effects  into  account. 


7.0  SUMMARY 

In  this  paper,  we  have  presented  metrics  for  evaluating  the  performance  of  single  array  acoustic  systems 
used  for  the  localization  of  small  arms  fire.  A  good  metrics  framework  does  not  necessarily  provide  a  true 
account  of  the  performance  of  a  system  in  actual  operational  conditions.  Rather,  it  provides  a  means  of 
quantifying  performance  that  can  be  reproduced  through  a  reasonable  effort. 

Small  arms  fire  produces  a  shockwave  and  a  muzzle  blast,  along  with  reflections  and  reverberation. 
Recognizing  that  a  shot  has  been  fired  is  accomplished  by  recognizing  the  shockwave.  Because  the  time 
features  of  the  shockwave  fade  away  with  range,  the  probability  of  detection  depends  on  average  upon  the 
miss  distance.  Recognizing  the  presence  of  a  clear  muzzle  blast  can  help  in  recognizing  that  a  shot  was 
fired,  but  that  is  a  rare  event.  Shooters  are  often  far  from  the  sensor  and  may  be  firing  from  among  or 
within  buildings. 

The  performance  metrics  are  based  on  defining  error  bounds  for  each  parameter  of  the  solution:  range, 
azimuth  angle,  elevation  angle,  miss  distance  and  calibre.  A  solution  is  declared  to  be  good  if  the  error  on 
each  of  those  parameters  is  within  the  predefined  bounds.  The  probability  of  good  solution  is  the 
probability  that  a  solution  reported  to  the  operator  will  be  within  the  specified  bounds  for  all  the 
parameters. 

The  probability  of  false  alarm  has  little  meaning  in  the  context  of  small  arms  fire  localization.  If  the 
background  noise  was  truly  random,  one  could  probably  set  the  algorithms  to  meet  a  certain  probability  of 
false  alarm.  For  an  acoustic  system  operating  outdoors,  ambient  sounds  are  not  random  but  a  sum  of 
recognizable  noises:  people  talking,  door  slamming,  engine  starting,  traffic  noise,  etc.  One  of  those  noises 
may  cause  false  alarms.  If  this  is  known  to  the  manufacturer,  operators  should  be  informed.  For  example, 
if  false  alarms  occur  when  people  talk  within  one  metre  of  the  sensor,  the  operator  should  be  told  how  to 
avoid  causing  false  alarms. 
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V 


Introduction 


Acoustic  Small  Arms  localization  Systems 
(SADLS)  are  being  introduced  in  operation 

SADLS  are  being  included  in  the  requirements  for 
new  combat  vehicles 

Standard  metrics  are  required 

-  to  specify  the  performance 

-  to  evaluate/compare  systems 
DRDC  Valcartier  has  been  involved  with 

-  Ferret 

-  and  its  procurement 
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Acoustic  Phenomenology 


angle  of  fire 


weapon 
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Acoustic  Phenomenology  ... 
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Acoustic  signals:  theoretical  and  real 


shockwave 


Shockwave 

Reflections 


muzzle  blast 


reverberation 


ambient  Noise 


muzzle  blast  reflection 


0.2  SECOND 
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Performance  Metrics 


Probability  of  detection 
Probability  of  full  localization 
Probability  of  good  solution 

>  Azimuth  angle  accuracy 

>  Range  accuracy 

>  Miss  distance  accuracy 

>  Caliber  accuracy 
Probability  of  false  alarm 
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Performance  Metrics 


•  •  • 


•  Intended  for: 


-  Open  terrain  . .  .or  a  well-defined  environment 

-  Single  array  systems 


-  Single  shot 

-  Multiple  shots  spaced  in  time  by  a  few  seconds 


Each  metric  proposed  should  be  measurable  under 
pre-defined  conditions 
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f\$J  Probability  of  detection 


Probability  that  a  shot  is  detected  and  recognized 

-  A  shockwave  and  a  muzzle  blast  are  produced 

-  Muzzle  blast  can  not  be  trusted 

-  Shockwave  is  used  for  detection 

-  Shockwave  recognition  depends  upon  range 

•  Turbulence  and  refraction 


We  propose  to  specify  the  probability  of  detection  at 
a  number  of  given  ranges 
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Probability  of  full  localization 


Probability  that  a  full  solution  is  provided  given  that 
the  shot  is  detected  and  recognized 

-  Full  solution:  range,  azimuth,  elevation,  miss 
distance  and  caliber 

-  Probability  of  recognizing  the  muzzle  blast 


Some  range  dependency 

Refraction,  turbulence,  shooter  environment 


We  propose  to  use  a  single  value  for  the  probability 
of  full  localization,  independent  of  range 
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Probability  of  good  solution 

or  confidence  interval 


Probability  of  meeting  the  azimuth  angle  accuracy, 
elevation  angle  accuracy,  range  accuracy,  miss 
distance  accuracy  AND  calibre  accuracy. 

-  Intended  for  the  operator 

-  It  answers  the  question: 

How  much  trust  can  I  put  in  the  output  of  the 
system 
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Probability  of  false  alarm 


Probability  of  being  wrongly  alerted  to  a  small  arms 
fire 

-  False  alarms  are  created  by  specific  sounds  that 
mimic  the  shockwave 


-  Question:  What  are  the  chances  of  being  in  the 
presence  of  that  specific  sound? 

-  Friendly  fires  in  the  vicinity  of  the  system 
should  be  filtered  out 

We  propose  that  the  manufacturer  indicates  to  the 
user  the  sounds  that  cause  false  alarms 
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Azimuth  angle  accuracy 


Bounds  that  defines  the  range  of  azimuth  angle  for  a 
good  solution 

-  Azimuth  angle  errors  come  from 

•  Array  misalignment/position 

•  Atmospheric  turbulence 
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Elevation  Angle  Accuracy 


Bounds  that  defines  the  range  of  elevation  angle  for 
a  good  solution 

-  Elevation  angle  errors  come  from 

•  Array  misalignment/position 

•  Atmospheric  refraction  and  turbulence 

•  Can  not  be  trusted  at  long  ranges 
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Range  accuracy 


Bound  that  defines  the  range  of  shooter  distances  for 
a  good  solution 

-  Range  errors  come  from 

•  Atmospheric  turbulence  »>  angles 


Muzzle  blast  recognition 
Range  errors  increase  with  range 


Range  accuracy  can  be  specified  for  short  ranges 
and  long  ranges 
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Miss  distance  accuracy 


Bound  that  defines  the  range  of  miss  distances  for  a 
good  solution 

-  Miss  distance  errors  come  from 

•  Atmospheric  turbulence/absorption 


In  principle,  miss  distance  errors  increase 
with  miss  distance 


We  propose  to  use  a  single  value,  independent  of 
miss  distance 
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Caliber  accuracy 


Bound  that  defines  the  range  of  caliber  for  a  good 
solution 

-  Caliber  errors  come  from 

•  Atmospheric  turbulence/absorption 


In  principle,  caliber  errors  increase  with  miss 
distance 


We  propose  to  use  a  single  value,  independent  of 
miss  distance 
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Summary 


A  good  metrics  framework  does  not  necessarily 
provide  a  true  account  of  the  performance  of  a 
system  in  actual  operational  conditions. 

It  provides  a  means  of  quantifying  performance  that 
can  be  reproduced  through  a  reasonable  effort. 

•  The  probability  of  false  alarm  has  little  meaning 
because  the  system  noise  is  not  random 
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